Modeling the Evolution of Knowledge in Learning Systems

How do reasoning systems that learn evolve over time? What are the properties of different learning strategies? Characterizing the evolution of these systems is important for understanding their limitations and gaining insights into the interplay between learning and reasoning. We describe an inverse ablation model for studying how large knowledge-based systems evolve: Create a small knowledge base by ablating a large KB, and simulate learning by incrementally re-adding facts, using different strategies to simulate types of learners. For each iteration, reasoning properties (including number of questions answered and run time) are collected, to explore how learning strategies and reasoning interact. We describe several experiments with the inverse ablation model, examining how two different learning strategies perform. Our results suggest that different concepts show different rates of growth, and that the density and distribution of facts that can be learned are important parameters for modulating the rate of learning. Introduction and Motivation In recent years, there has been considerable interest in Learning by Reading [Barker et al 2007; Forbus et al 2007, Mulkar et al 2007] and Machine Reading [Etzioni et al 2005; Carlson et al 2010] systems. The study of these systems has mainly proceeded along the lines of measuring their efficacy in improving the amount of knowledge in the system. Learning by Reading (LbR) systems have also explored reasoning with learned knowledge, whereas Machine Reading systems typically have not, so we focus on LbR systems here. These are evolving systems: over time, they learn new ground facts and new predicates and collections are introduced, thereby altering the structure of their knowledge base (KB). Given the nascent state of the art, so far the learned knowledge is typically small compared to the knowledge base the system starts with. Hence the learning trajectory and final state of the system is known for all practical purposes. But what will be the learning trajectory as the state of the art improves, and the Copyright © 2012, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. number of facts the system has learned by reading (or using machine reading techniques) dwarfs its initial endowment? To explore such questions, we introduce an inverse ablation model. The basic idea is to take the contents of a large knowledge base (here, ResearchCyc 1 ) and make a simulation of the initial endowment of an LbR system by removing most of the facts. Reasoning performance is tested on this initial endowment, including the generation of learning goals. The operation of a learning component is simulated by gathering facts from the ablated portion of the KB that satisfy the learning goals, and adding those to the test KB. Performance is then tested again, new learning goals are generated, and the process continues until the system converges (which it must, because it is bounded above by the size of the original KB). This model allows us to explore a number of interesting questions, including: (1) How does the growth in the number of facts affect reasoning performance? (2) How might the speed at which different kinds of concepts are learned vary, and what factors does that depend upon? (3) Is learning focused, or are we learning facts about a wide range of predicates and concepts? (4) What are the properties of different learning strategies? (5) How does the distribution of facts that can be acquired affect the learning trajectory? The inverse ablation model provides a general way to explore the evolution of knowledge bases in learning systems. This paper describes a set of experiments that are motivated by LbR systems. Under the assumptions described below, we find that (1) the size of the KB rapidly converges, (2) the growth is limited to a small set of concepts and predicates, spreading to only about 33% of the entire growth possible, (3) different concepts show different rates of growth, with the density of facts being an important determining factor, and (4) Different learning strategies have significant differences in their performance, and the distribution of facts that can be learned also plays an important role. The rest of this paper is organized as follows: We start by summarizing related work and the conventions we assume for representation and reasoning. A detailed 1 http://research.cyc.com